IDENTIFICATION AND PLANNING SYSTEM AND METHOD FOR FILLING ORDERS

专利摘要:
identification and planning system and order fulfillment method. an identification and planning system includes a scanner set, a robotic manipulator and a control server. the control server detects a first plurality of inventory items in a first inventory storage system and determines a first set of position parameters for a first inventory item from the first plurality of inventory items detected, with respect to a position of one or more image sensors in the scanner assembly. the control server generates a plurality of trajectory selection plans for the first stock item, where each trajectory selection plan corresponds to a transformation of the first set of position parameters into a second set of position parameters in relation to to the robotic manipulator. the control server additionally selects a first trajectory selection plane from the plurality of trajectory selection planes and controls the robotic manipulator to select the first stock item from the first stock storage system, based on the first selection plane trajectory.
公开号:BR102019022468A2
申请号:R102019022468-1
申请日:2019-10-25
公开日:2020-05-26
发明作者:Anirudh Singh Shekhawat；Sameer Narkar；Akash Patil；Avilash Kumar；Vaibhav Tolia
申请人:Grey Orange Pte, Ltd.；
IPC主号:

专利说明:

“IDENTIFICATION AND PLANNING SYSTEM AND METHOD FOR FILLING ORDERS”
CROSS REFERENCE TO ORDERS RELATED BY REFERENCE
[0001] None.
FIELD OF THE INVENTION
[0002] Several types of disclosure are related to deposit automation technology. More specifically, several types of disclosure refer to a system and method of identification and planning for order fulfillment.
FUNDAMENTALS
[0003] Advances in warehouse automation technologies have driven the development of advanced goods-to-person (GTP) systems for automated selection, classification and / or withdrawal of different types of stock items for multi-line orders at distribution centers and attendance of a deposit. Conventional GTP systems include a scanner, such as a camera, which is usually mounted on a robotic manipulator. The scanner detects and identifies different stock items in a storage unit. Whenever the scanner scans an inventory item, the robotic manipulator waits for the scan to complete and subsequently generates an instruction to maneuver a robotic arm to select a scanned inventory item. This causes a delay in the response of the robotic manipulator to maneuver the robotic arm and select the scanned stock item. This further increases the cycle time for selecting inventory items for one or more orders. In addition, in a case where there is a demand for frequent shipments and shorter delivery times, the output of a conventional GTP system or a warehouse automation system suffers based on the variability of the throughput or even with a drop in transfer rate within and between shifts due to human factors. This leads to
Petition 870190138461, of 12/23/2019, p. 4/76
2/59 fulfillment of orders prone to errors and inflated costs of selecting and processing orders in the supply chain, which can be undesirable.
[0004] Other limitations and disadvantages of conventional and traditional approaches will become evident to one skilled in the art, by comparing the systems described with some aspects of the present disclosure, as established in the remainder of this application and with reference to the drawings.
SUMMARY
[0005] An identification and planning system and method for fulfilling orders is substantially shown and / or described in connection with at least one of the figures, as more fully set out in the claims.
[0006] In one embodiment, an identification and planning system is provided. The system includes a scanner set, a robotic manipulator and a control server. The scanner assembly includes one or more image sensors. The robotic manipulator includes a robotic arm and an end effector detachable to the robotic arm. The control server includes control circuits, which are configured to detect, by one or more image sensors, a first plurality of stock items in a first stock storage system. The control circuit is configured to determine a first set of position parameters for a first stock item from the first detected plurality of stock items, with respect to the position of an image sensor of the one or more image sensors in the set the scanner. The control circuit is configured to generate a plurality of trajectory selection plans for the first stock item. Each trajectory selection plane of the plurality of trajectory planes corresponds to a transformation of the first determined set of position parameters into a second set of position parameters
Petition 870190138461, of 12/23/2019, p. 5/76
3/59 in relation to the robotic manipulator. The control circuit is configured to select a first path selection plane from the plurality of path selection planes. The first path selection plan is selected before a selection operation is carried out on the first inventory item. The selection of the first path plane is based on the success of the selection operation in a test computer simulation, based on the second set of position parameters corresponding to the first selected path selection plane. The control circuit is configured to, in the selection operation, control the robotic arm and the end effector to select the first stock item of the first stock storage system, based on the first trajectory selection plan.
[0007] In one embodiment, an identification and planning system is provided. The system includes a control server. The control server includes control circuits, which are configured to detect a first plurality of inventory items in a first inventory storage system. The control circuit is configured to determine a first set of position parameters for a first stock item from the first detected plurality of stock items, with respect to the position of an image sensor of one or more image sensors. The control circuit is configured to generate a plurality of trajectory selection plans for the first stock item. Each trajectory selection plane of the plurality of trajectory selection planes corresponds to a transformation of the first set of position parameters into a second set of position parameters in relation to the robotic manipulator. The control circuit is configured to select a first path selection plane from the plurality of path selection planes. The first path selection plan is selected before a selection operation is carried out on the first inventory item. The selection of the first trajectory selection plane is based on the
Petition 870190138461, of 12/23/2019, p. 6/76
4/59 success of the selection operation in a test computer simulation, based on the second set of position parameters corresponding to the first trajectory selection plane. The control circuit is configured to, in the selection operation, control the robotic arm and the end effector detachably connected to the robotic arm to select the first stock item of the first stock storage system, based on the foreground plan. trajectory selection.
[0008] In one embodiment, a method of an identification and planning system is provided that includes control circuits and one or more image sensors. In the method, a first plurality of stock items in a first stock storage system is detected by one or more image sensors. A first set of position parameters is determined, by the control circuit, for a first stock item of the first detected plurality of stock items in relation to a position and an image sensor of one or more image sensors. A plurality of trajectory selection plans are generated by the control circuit for the first stock item. Each trajectory selection plane of the plurality of trajectory selection planes corresponds to a transformation of the first set of position parameters into a second set of position parameters in relation to the robotic manipulator. A first trajectory selection plane is selected, by the control circuit, from the plurality of trajectory selection plans. The first path selection plan is selected before a selection operation is carried out on the first inventory item. The selection of the first path selection plane is based on the success of the selection operation in a test computer simulation, based on the second set of position parameters corresponding to the first path selection plane. In the selection operation, a robotic arm and an end effector detachably connected to the robotic arm are controlled by the control circuit, to select the first stock item of the first
Petition 870190138461, of 12/23/2019, p. 7/76
5/59 stock storage system, based on the first trajectory selection plan selected.
[0009] In one embodiment, a non-transitory computer-readable storage medium is provided, which, when performed by at least one processor, causes at least one processor to execute the method of the identification and planning system described above.
[0010] These and other characteristics and advantages of the present disclosure can be appreciated from a review of the detailed description below of the present disclosure, together with the attached figures in which similar reference numbers refer to equal parts.
BRIEF DESCRIPTION OF THE FIGURES
[0011] FIG. 1 illustrates a block diagram of an identification and planning system for order fulfillment, according to a disclosure modality.
[0012] FIG. 2A illustrates a detailed block diagram of the identification and planning system of FIG. 1 for order fulfillment, according to a disclosure modality.
[0013] FIG. 2B illustrates a detailed block diagram of the identification and planning system for order fulfillment, according to an alternative form of disclosure.
[0014] FIG. 3 illustrates an exemplary deposit scenario comprising the identification and planning system of FIG. 1, according to a disclosure mode.
[0015] FIG. 4A illustrates an exemplary stock storage system placed in front of a scanner set for detecting stock items from the stock storage system, according to a disclosure modality.
[0016] FIG. 4B illustrates an exemplary robotic manipulator that performs a selection operation on different items of
Petition 870190138461, of 12/23/2019, p. 8/76
6/59 inventory in an inventory storage system based on an ideal trajectory selection plan, according to a disclosure modality.
[0017] FIG. 5A illustrates a first scenario for sequentially executing a scan operation, a selection operation and an order consolidation for inventory items accommodated in an inventory storage system, according to a disclosure modality.
[0018] FIG. 5B illustrates a second scenario for the scheduled execution of the scanning operation and the selection operation for a row of different stock storage systems, according to a disclosure modality.
[0019] FIGS. 6A and 6B, collectively, illustrate a scenario for the execution of a placement operation by an exemplary robotic manipulator in two different stages, according to a disclosure modality.
[0020] FIGS. 7A, 7B and 7C collectively illustrate a flow chart that describes a method for fulfilling orders in a warehouse, according to a disclosure modality.
DETAILED DESCRIPTION
[0021] The implementations described below can be found in a system and method of identification and planning disclosed for order fulfillment. The disclosed identification and planning system provides a solution to minimize the loss of productivity and optimize the total cycle time in the selection of different types of stock items (for example, fast moving consumer goods (FMCG), durable goods or others consumer goods) from stock storage systems. The identification and planning system disclosed includes a scanner set and a robotic manipulator, which are spatially located in different positions in a warehouse. As a result of such positioning of the scanner and manipulator assembly
Petition 870190138461, of 12/23/2019, p. 9/76
7/59 robotic, optimization can be achieved based on the execution of a scan operation and trajectory selection planning before a selection operation by the robotic manipulator. The path planning operation can correspond to the estimation of the ideal position values (that is, an ideal path selection plan or a placement path plan along the horizontal, vertical and rotational axes) for the robotic manipulator. These ideal position values can be used to control the robotic manipulator to perform the selection or placement operation. In the selection operation (or placing operation), an end effector of the robotic manipulator can be operated based on the ideal position values for selecting (or placing) an item (for example, an ordered product) from (or for) a storage shelf (or pallet) of a storage system in the vicinity of the robotic manipulator
[0022] In addition, the disclosed solution determines an ideal trajectory selection plan before the stock storage system reaches the robotic manipulator for the selection operation. This further optimizes overhead overhead costs, such as order selection time and robotic manipulator response time, in order fulfillment. The determination of the trajectory selection plane is done in a simulated environment, where different position and gripping parameters of the robotic manipulator are recursively modified to identify the ideal position and gripping parameters. The application of the ideal position and the gripping parameters can result in the success of the selection operation in each attempt to select a stock item. In addition, the success of the selection operation can be attributed to the implementation of a GTP system. Compared to a Person to Merchandise (PTG) system, where an individual worker must manually approach a storage system, select and still drop the stock item into dedicated bags, the use of a GTP system helps with the planned organization
Petition 870190138461, of 12/23/2019, p. 10/76
8/59 for stock items in different storage systems and a planned support of several mobile robots to move the stock items. This planned organization of inventory items helps to predict inventory movement and control over the order fulfillment rate in real time or near real time. The planned organization of different inventory items can be achieved based on historical demand, current demand and demand forecasts for different inventory items. There may not be a requirement to process trajectory selection plans at the end of the robotic manipulator. Thus, in the disclosed solution, a lower cycle time for order selection can be achieved in comparison with conventional GTP selection systems.
[0023] The disclosed solution also provides a queuing stage. In the queuing stage, an inventory storage system can be placed for a selection operation in a queue in parallel with a scan operation for another inventory storage system. As the sweep operation and the path selection planning operation occur before the selection operation, the separation time is significantly reduced compared to the selection time of conventional GTP systems. Therefore, the disclosed solution provides greater productivity compared to conventional GTP systems. Since the scan operation takes place before the selection operation, only automatically selectable inventory items can be selected for the selection operation. This can ensure accuracy in order fulfillment and ensure that only orders containing all automatically selectable inventory items are approved for autonomous selection. Orders containing manually selectable inventory items can be processed through an order consolidation stage, where these inventory items are selected and handled manually by warehouse workers. The order consolidation process can be performed before the selection operation by the ordering system
Petition 870190138461, of 12/23/2019, p. 11/76
9/59 identification and planning.
[0024] As a conventional deposit can handle high transfer rate variability within and between shifts due to human factors, the proposed solution provides a way to maintain a consistent transfer rate and order fulfillment without errors, with lower cost per shipment which offers companies a competitive advantage over competitors. The proposed solutions also enhance the capabilities of conventional GTP systems to run multiple shifts to meet peak demand cycles and improve overall supply chain efficiency and productivity.
[0025] FIG. 1 illustrates a block diagram of an identification and planning system for order fulfillment, according to a disclosure modality. In FIG. 1, a block diagram 100 of an identification and planning system 102 is shown. The identification and planning system 102 includes a control server 104, a plurality of stock storage systems 106A ,. . . 106N (which includes a plurality of storage compartments 108A,..., 108N), a plurality of mobile robots 110A ,. . . , 110N and a scanner assembly 112 (which includes a plurality of image sensors 114). The identification and planning system 102 further includes a robotic manipulator 116 (which includes a robotic arm 118 and an end effector 120) and a human machine interface (HMI) 122. Also shown is a communication network 124 that is communicatively coupled to the control server 104, the plurality of stock storage systems 106A ,. . . , 106N, the plurality of mobile robots 110A ,. . . , 110N, scanner assembly 112, robotic manipulator 116 and HMI 122. In addition, a user 126 associated with HMI 122 is shown.
[0026] The identification and planning system 102 can comprise logic, appropriate circuits and interfaces that can be configured to control the selection, storage, removal or replacement
Petition 870190138461, of 12/23/2019, p. 12/76
10/59 of different inventory items in one or more warehouse fulfillment and distribution centers (an example of a warehouse scenario is shown in FIG. 3). The identification and planning system 102 can be configured to manage a robotic goods-to-person (GTP) facility, where different inventory items are automatically removed from stock storage systems (for example, mobile storage units (MSUs)) and supplied to selection stations or transferred to depot packaging stations. The identification and planning system 102 can include one or more control servers (for example, control server 104) that can control different operations, such as detecting inventory items, planning a selection plan (that is, a strategy and selection) for the robotic manipulator 116 and execution of the trajectory selection plan to select specific stock items.
[0027] The control server 104 can comprise appropriate logic, circuits and interfaces that can be configured to control the execution of different operations associated with the removal or selection of different stock items from one or more stock storage systems in a warehouse. service and distribution of the deposit. In addition, control server 104 can be configured to control the execution of different operations associated with replenishment (for example, a robotic manipulator placement operation 116) of stock storage systems, order consolidation for different types of items inventory to be selected in a cycle or batch of orders, an order classification operation, palletizing and / or depalletizing of inventory items and the like. Some of the operations may include, but are not limited to, stock profile, planning and checking stock item selection and stock selection and stock transfer to stock pick stations. Control server 104 can be part of an automated warehouse management system (WMS), which can be a system
Petition 870190138461, of 12/23/2019, p. 13/76
11/59 independent. Alternatively, control server 104 can be integrated with automated WMS. In addition, in certain cases, control server 104 can be further integrated with supply chain systems and / or integrated enterprise resource planning (ERP) systems. As an example, control server 104 can be a centralized cloud server (private or shared), a part of a warehouse data center, or a cluster of internal on-premises servers, where each server can be dedicated to a specific operation of the identification and planning system 102.
[0028] The plurality of stock storage system 106A ,. . . 106N can be physical storage units, where each physical storage unit can include a plurality of storage compartments 108A ,. . . , 108N. Each storage compartment of the plurality of storage compartments 108A ,. . . , 108N can be present in different spatial positions in an inventory storage system to accommodate a plurality of inventory items. Examples of a stock storage system may include, but are not limited to, multilayer shelves, pallet shelves, shelves, mobile shelves, mezzanine floors, vertical lifting modules, horizontal carousels, conveyors and vertical carousels. In certain embodiments, the plurality of stock storage systems 106A ,. . . 106N can correspond to mobile storage units from one storage space to another storage space in the warehouse. In such implementations, the movement of the plurality of stock storage systems 106A ,. . . 106N can be activated by the plurality of mobile robots 110A ,. . . , 110N.
[0029] The plurality of mobile robots 110A ,. . . , 110N can be autonomous mobile robots (AMRs), autonomous guided vehicles (AGVs) or a combination of them, at call centers and depot distribution centers. The plurality of mobile robots 110A ,. . . , 110N can
Petition 870190138461, of 12/23/2019, p. 14/76
12/59 include appropriate logic, circuits and interfaces that can be configured to automate pickup, storage, replenishment and selection of payloads (for example, stock storage systems or pallet of stock items) at a service and distribution center. a deposit. For the identification and planning system 102, each mobile robot from the plurality of mobile robots 110A ,. . . , 110N can be configured to allow (as part of a GTP configuration) the movement of a payload from stock storage areas to different scans and selection areas (for example, a queuing station for order selection) reserved for the scanner set 112 and robotic manipulator 116.
[0030] In some embodiments, each mobile robot of the plurality of mobile robots 110A ,. . . , 110N can include different functional components, such as a lifting mechanism, an adaptive payload management system and a stand-alone guidance system, by the use of which a payload (for example, a stock storage system or stock pallet) ) can be moved to different locations in the warehouse. Each mobile robot can be equipped with suitable components to allow a multi-storey goods transfer, for example, a mobile robot can move within different floors and meet the requirements of the identification and planning system 102, choosing different single-storey MSUs and transferring to a selective pallet rack (PPS) system. The PPS can correspond to an area in the warehouse where inventory items are removed or selected by human operators before entering the shelves or after selecting the exit from the shelves of different stock storage systems.
[0031] In addition, each mobile robot can be configured to adapt to different functional parameters, for example, payload weight, transfer path, cycle time, etc. , according to the continuous change in stock profiles, demand patterns and peak prices
Petition 870190138461, of 12/23/2019, p. 15/76
13/59 orders. Each mobile robot of the plurality of mobile robots 110A ,. . . , 110N can be functionally the same or different from each other, with possible variations in payload capacity (in pounds (pounds) or kilograms (kg)).
[0032] Scanner set 112 may correspond to a three-dimensional (3D) camera system for detecting and / or identifying different stock items in a stock storage system of the plurality of stock storage systems 106A ,. . . , 106N. As part of the 3D camera system, scanner set 112 can include the plurality of image sensors 114 that can be configured to capture a plurality of images of different stock items, along with detailed information of the different stock items in a system stock storage. In addition, in certain embodiments, scanner assembly 112 includes an orientation mechanism (a programmable or computer-controlled actuator) and a orientable portion (for example, an actuator belt, a plate, etc.) mounted on a frame. Support. The plurality of image sensors 114 can be fixed in the orientable portion and actuated in different directions (for example, along orthogonal axes in a plane) using the orientation mechanism. In such implementations, a position of an image sensor in scanner assembly 112 at the time of scanning an inventory item may correspond to a position of the inventory item in the inventory storage system.
[0033] According to one embodiment, the plurality of image sensors 114 can be part of the 3D camera system installed in the scanner set 112. Each image sensor can be one of a 3D range camera or a stereo vision camera 3D that can be affixed to the guide part of the scanner assembly 112. The plurality of image sensors 114 can comprise logic, circuitry and suitable interfaces that can be configured to capture a plurality of 3D images of a plurality of stock items stored in different compartments
Petition 870190138461, of 12/23/2019, p. 16/76
14/59 stock storage system storage. The 3D image can include RGB-D information from a scene in the field of view of a corresponding image sensor from the plurality of 114 image sensors. In RGB-D information, RGB can correspond to the red, green and blue pixel values, while D can correspond to item depth information of a stock in the FOV of a corresponding image sensor. Depth information can include values of distance from a focal plane of the image sensor to different points in the stock item. Examples of implementing the plurality of image sensors 114 may include, but are not limited to, a flight time camera system (TOF), a stereo camera system, a 3D structured light scanner that uses projected light patterns, a TOF sensor point combined with a scanning mechanism, such as Light Detection and Variation (LiDAR), a matrix TOF camera, a rotating pulsed light scanner, a pulsed light TOF scanner and a modulated light TOF scanner.
[0034] The robotic manipulator 116 can correspond to a manipulator system that can be configured to perform different operations, such as: selecting, holding, grabbing, transferring, sorting, removing or putting back stock items to / from the storage system of stock. Robotic manipulator 116 can include robotic arm 118 coupled to end effector 120. In certain embodiments, end effector 120 can be a gripping or holding tool detachably attached to robotic arm 118. Details of such an implementation are described in detail, for example, in FIG. 4B.
[0035] Robotic arm 118 may include different functional portions (for example, arms), the movement of which can be maneuvered by a plurality of actuators to guide end effector 120 to a specific stock item, which can be detected by the scanner assembly 3D camera 112. The movement of robotic arm 118 in 3D space can be restricted by a defined number of degrees of
Petition 870190138461, of 12/23/2019, p. 17/76
15/59 freedom displayed by different functional parts of the robotic arm 118. Different functional portions can be maneuvered based on machine instructions received from control server 104 based on ideal position parameters of different stock items in relation to a position ( i.e., an origin) of the robotic manipulator 116.
[0036] The end effector 120 can be a tool, a set or an apparatus that can be removably attached to one or more free joints in the robotic arm 118. The end effector 120 can be configured to apply different effect mechanisms by use pneumatic systems, hydraulic systems or electromechanical systems, to select, hold, grasp, slide, throw, press, shrink or apply other physical effects to different types of stock items in the plurality of the stock storage system 106A ,. . . , 106N, based on the identification of the stock items. Examples of end effector 120 may include, but are not limited to, robotic clamps, such as electric clamps, vacuum clamps (such as suction cups), pneumatic clamps, magnetic clamps, and robotic fingers. In certain embodiments, end effector 120 can be coupled to an automatic tool changer, which can act as an interface between a flange of robotic arm 118 (next to a free end of robotic arm 118) and end effector 120.
[0037] The HMI 122 can understand logic, circuits and suitable interfaces that can be configured to display a simulation of different operations of the identification and planning system 102 in real time or in near real time. The simulation can be an interactive 2D or 3D simulation (for example, interactive for the user) of different functional components, such as scanner set 112 and robotic manipulator 116, in the warehouse. In the simulation, real-time movements and prediction for subsequent movements of different functional components can be simulated further. HMI 122 can include a monitor, a processing circuit, a network interface, an input / output (I / O) interface
Petition 870190138461, of 12/23/2019, p. 18/76
16/59 and / or other circuits.
[0038] The communication network 124 can comprise logic, circuit and suitable interfaces that can be configured to provide a plurality of network ports and a plurality of communication channels for transmission and reception of data related to the operations of the identification and planning system 102. Each network port can correspond to a virtual address (or a physical address of the machine) for transmitting and receiving communication data. For example, the virtual address can be an IPV4 (Internet Protocol Version 4) (or an IPV6 address) and the physical address can be a MAC (Media Access Control) address. Communication network 124 can be associated with an application layer for implementing communication protocols based on one or more communication requests from at least one of the one or more communication devices (for example, I / O or HMI devices 122). Communication data can be transmitted or received using communication protocols. Examples of communication protocols may include, but are not limited to, HTTP (Hypertext Transfer Protocol), FTP (File Transfer Protocol), SMTP (Simple Mail Transfer Protocol), DNS (Domain Network System) and CMIP (Common Management Interface Protocol).
[0039] According to one embodiment, the communication data can be transmitted or received through at least one communication channel of the plurality of communication channels in the communication network 124. The communication channels can include, but are not limited to a wireless channel, a wired channel, a combination of wireless and wired channel it. The wireless or wired channel can be associated with a data standard that can be defined by a local area network (LAN), a personal network (PAN), a wireless local area network (WLAN), a wireless sensor network ( WSN), wireless area network (WAN) and wireless wide area network (WWAN).
[0040] In operation, at a given moment, the control server 104 can be configured to receive information from
Petition 870190138461, of 12/23/2019, p. 19/76
17/59 orders associated with the fulfillment of one or more orders from an order management server (not shown). The order management server can be part of a warehouse supply chain system or it can be integrated with an ERP system implemented at the warehouse level or at the warehouse cluster level. Order information can include, but is not limited to, product information (such as descriptions, attributes, locations, order quantities), available stock for promise (ATP) and supply details, supplier information, purchasing information, details about entry of orders and consumer services (for example, returns and refunds) and payment information (such as credit cards, billing and payment verification).
[0041] The control server 104 of the identification and planning system 102 can be configured to analyze the order information and generate a list (or several lists) of stock items to be selected in a cycle or batch operation (or several cycles) to fulfill one or more orders. In order to select and remove inventory items specified in the received list, the control server 104 can be configured to instruct one or more mobile robots from the plurality of mobile robots 110A ,. . . , 110N, deployed in different areas of the warehouse, to reach different storage storage systems that have the inventory items specified in the received inventory item list. The instruction can also include information associated with specific locations (in the warehouse) where different stock storage systems can be moved by one or more mobile robots. These locations can correspond to dedicated selection stations, where inventory items from different inventory storage systems can be selected (or sorted) and removed (or transferred) to dedicated bags or shelves for subsequent pipeline operations for one or more orders.
[0042] According to the instructions, the one or more mobile robots can be configured to lift and move a
Petition 870190138461, of 12/23/2019, p. 20/76
18/59 stock storage of the plurality of stock storage systems 106A ,. . . , 106N, from a first location (such as an inventory storage area) to a location associated with a sorting station. The selection station may include scanner assembly 112 and robotic manipulator 116 present in a specific region of the selection station. Alternatively, multiple sets of scanners and several robotic manipulators can be present in a row or in a specific arrangement to select multiple inventory items from different inventory storage systems at a time.
[0043] The plurality of mobile robots 110A ,. . . 110N can include a first mobile robot 110A which can be configured to move a first stock storage system 106A in a dedicated queue for scanner assembly 112 and robotic manipulator 116 at the selection station. According to one embodiment, there may be different selection stations in the warehouse, with each selection station having the scanner set 112 and the robotic manipulator 116.
[0044] The queue can correspond to regions of the grid that can act in a dedicated path for the movement of different stock storage systems in the selection station. The grid of regions can correspond to a plurality of locations marked by a fiducial marker, which can be a physical fiducial marker or a virtual fiducial marker. The first mobile robot 110A can be configured to move the first stock storage system 106A through the grid regions, where each fiducial marker acts as a stopping place for the first mobile robot 110A. In certain scenarios where the queue may be empty, that is, unoccupied by the different storage systems, each fiducial marker can be ignored as a stopping place by the first mobile robot 110A. It should be understood that the scope of the disclosure cannot be limited to the use of fiducial markers. In some modalities, the fiducial marker cannot be used and other systems for identifying
Petition 870190138461, of 12/23/2019, p. 21/76
19/59 position can be used. For examples of such position identification systems, they may include, but are not limited to, internal position system (IPS), Wi-Fi positioning system (WPS) or other offline or online geographical location detection systems.
[0045] According to one embodiment, the scanner assembly 112 and the robotic manipulator 116 can be located side by side and can be placed in front of different regions of the grid. The control server 104 can be configured to perform, using the plurality of image sensors 114 in the scanner assembly 112, a scanning operation on a first plurality of stock items in the first stock storage system 106A. In the scanning operation, the plurality of image sensors 114 can be configured to capture a plurality of 3D images of the first plurality of stock items. Each 3D image can include RGB-D information about one or more stock items in the field of view of an image sensor from the plurality of image sensors 114. RGB-D information can be used by control server 104 to detect the first plurality of stock items in the first stock storage system 106A.
[0046] The control server 104 can be further configured to extract color information and depth information from a 3D image that includes the first stock item. The 3D information can be estimated by the control server 104 for the first stock item. The 3D information can correspond to different surface attributes of the first stock item in the first stock storage system 106A, which can be obtained from a scan of the first plurality of stock items. The control server 104 can be further configured to determine a first set of position parameters for a first stock item from the first detected plurality of stock items, with respect to the position of an image sensor from the plurality of image sensors 114 in the scanner assembly
Petition 870190138461, of 12/23/2019, p. 22/76
20/59
112. The first set of position parameters can correspond to the position values of the first stock item in terms of 3D Cartesian coordinates. To select the first stock item, a selection plan may be required to specify different parameters related to the movement of the robotic arm 118 and end effector 120, together with the holding mechanism of the first stock item.
[0047] The control server 104 is configured to generate a plurality of trajectory selection plans for the first stock item. Each trajectory selection plane of the plurality of trajectory selection planes can correspond to a transformation of the first determined set of position parameters into a second set of position parameters in relation to robotic manipulator 116 (i.e., an origin of the robotic manipulator) 116). Alternatively, each trajectory selection plane can include a set of position values and other grab parameters that can be used by the control server 104 to simulate a robotic manipulator maneuver to select (or remove) items from stock (or to ) storage shelves of a storage system. The first set of position parameters can be spatially different from the second set of position parameters. The spatial differentiation can be caused by the difference in the position of the scanner assembly 112 and the robotic manipulator 116.
[0048] The plurality of trajectory selection plans can be generated by applying a plurality of transformation operations on the first set of position parameters to obtain the second set of position parameters. The plurality of transformation operations can correspond to iterative changes in the first set of position parameters (that is, position values measured in relation to the position of an image sensor).
[0049] In order to select an ideal trajectory selection plane (ie a first trajectory selection plane), a
Petition 870190138461, of 12/23/2019, p. 23/76
21/59 test computer simulation for each path selection plan of the plurality of path selection plans can be performed by the control server 104. The test computer simulation can be further visualized on HMI 122 connected communicatively to the server control system 104, via communication network 124. In the test computer simulation, the control server 104 can be configured to verify a successful selection operation corresponding to the application of each selection plan from the generated plurality of selection plans. trajectory. With each transformation operation, the position parameters for the robotic manipulator 116 can be adjusted and rendered further in a simulation to verify that the robotic arm 118 and the end effector are able to reach and hold the first stock item of the first system 106A stock storage. It can be verified that the end effector 120 (for example, a pneumatic clamp) is able to successfully grab and hold the first item in stock for a necessary period, without falling or damaging the packaging or the product kept as the first item of the stock.
[0050] The control server 104 can be further configured to select a first trajectory selection plane from the generated plurality of trajectory selection plans. The first trajectory selection plane is selected before a selection operation is performed on the first stock item. According to one embodiment, the control server 104 can be configured to select the first path selection plane for the first inventory item during an idle state of scanner assembly 112 and robotic manipulator 116. The inactive state may correspond to a state in which scanner assembly 112 and robotic manipulator 116 may be in the non-operating state for a specific period of time. Therefore, the first trajectory selection plan can be selected at a time when the order volume is low to save time when the order volume can increase.
[0051] The selection of the first plan for the selection of
Petition 870190138461, of 12/23/2019, p. 24/76
22/59 trajectory can be based on the success of the selection operation in the test computer simulation. Success can be determined based on the second set of position parameters corresponding to the first path selection plane. As the ideal path selection plan is selected, the control server 104, in the selection operation, can be further configured to control the robotic arm 118 and end effector 120 to select the first stock item of the first storage system 106A, based on the first trajectory selection plan. More specifically, in the selection operation, the robotic manipulator 116 can further control a plurality of actuators on the robotic arm 118 to allow movement of the robotic arm 118 and end effector 120 within the defined number of degrees of freedom in 3D space. The operation of the identification and planning system is further described in detail, for example, in FIGs. 2A, 2B, 3, 4A, 4B, 5A and 5B.
[0052] FIG. 2A illustrates a detailed block diagram of the identification and planning system of FIG. 1 for order fulfillment, according to a disclosure modality. FIG. 2A is explained in conjunction with the elements of FIG. 1. With reference to FIG. 2A, a block diagram 200A of the identification and planning system 102 is shown. In block diagram 200A, an embodiment of the identification and planning system 102 is shown that includes the control server 104 and the plurality of mobile robots 110A ,. . . , 110N. The identification and planning system 102 further includes scanner assembly 112 (which includes the plurality of image sensors 114) and robotic manipulator 116 (which includes robotic arm 118 and end effector 120). The control server 104 includes a network interface 202, an input / output (I / O) interface 204, memory 206 and control circuit 208.
[0053] The network interface 202 can comprise logic, circuits and suitable interfaces that can be configured to establish and allow communication between control server 104 and different
Petition 870190138461, of 12/23/2019, p. 25/76
23/59 components of the identification and planning system 102, through the communication network 124. The network interface 212 can be implemented using several known technologies to support the wired or wireless communication of the control server 104 with the communication network. communication 124. Network interface 202 may include, but is not limited to, an antenna, a radio frequency (RF) transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, an encoder encoded chipset (CODEC), a SIM card (subscriber identity module) and a local buffer circuit.
[0054] The I / O interface 204 may comprise appropriate logic, circuit, interfaces and / or code that can be configured to receive user input and transmit server outputs through a plurality of data ports on the control server 104 The I / O interface 204 can comprise several input and output data ports for different I / O devices. Examples of such I / O devices may include, but are not limited to, a touchscreen, a keyboard, a mouse, a joystick, a microphone, an image capture device, a liquid crystal display (LCD) and / or a speaker screen.
[0055] Memory 206 can comprise logic, circuits and suitable interfaces that can be configured to store instructions executable by control circuits 208. In addition, memory 206 can be configured to receive and store (programmable) instructions from one or more servers or HMI 122, via communication network 124. Memory 206 may correspond to persistent storage and / or non-persistent storage of control server 104. Examples of memory 206 may include, but are not limited to, magnetic storage unit , a solid state drive, PROM (Programmable Read Only Memory), erasable PROM (EPROM), electrically EPROM (EEPROM), flash memory. In some embodiments, a set of centralized or distributed network of memory devices
Petition 870190138461, of 12/23/2019, p. 26/76
Peripheral 24/59 can interface with control server 104, as an example, on a cloud server.
[0056] The control circuit 208 may comprise logic, circuit and suitable interfaces that can be configured to control the execution of different operations associated with the selection, removal or classification of stock items different from the plurality of stock storage systems 106A ,. . . 106N in the tank. In addition, control circuit 208 can be configured to control the execution of different operations associated with the replenishment (that is, to replenish empty storage compartments) of stock items in empty (unallocated) storage compartments of the plurality of storage systems. 106A stock storage ,. . . , 106N. Control circuit 208 can also be configured to handle operations, such as order consolidation for different types of inventory items to be removed in a cycle or batch of orders, palletizing and / or depalletizing inventory items and the like. Some of the operations may include, but are not limited to, stock profile, planning and checking stock item selection and stock selection and stock transfer to stock pick stations. Examples of the 208 control circuit implementation may include, but are not limited to, application-specific integrated circuits (ASICs), system chips (SOCs), x86 / x64 processors, reduced instruction set (RISC) or complex instruction set architecture (CISC) of central processing units (CPUs), field programmable port arrays (FPGA) and programmable logic devices (PLDs) / complex PLDs.
[0057] FIG. 2B illustrates a detailed block diagram of the identification and planning system for order fulfillment, according to an alternative form of disclosure. FIG. 2B is explained together with the elements of FIGs. 1 and 2A. With reference to FIG. 2B, a block diagram 200B of the identification and
Petition 870190138461, of 12/23/2019, p. 27/76
25/59 planning 102. In block diagram 200B, another modality of the identification and planning system 102 is shown that includes only control server 104. Control server 104 can be a centralized cloud application server that can control operations the plurality of mobile robots 110A ,. . . , 110N, scanner assembly 112 and robotic manipulator 116, through multiple distribution centers and service of one or more depots. The identification and planning system 102 can be implemented as one of a software as a service (SaaS), platform as a service (PaaS) or infrastructure as a service (IaaS) for its own, self-managed, third-party or affiliated distribution and service centers. warehouses that can cater to different types of businesses, for example, electronic retail, electronics, clothing, grocery, freight forwarding, FMCG and the like.
[0058] FIG. 3 illustrates an exemplary deposit scenario comprising the identification and planning system of FIG. 1, according to a disclosure mode. FIG. 3 is explained in conjunction with the elements of FIGs. 1, 2A, and 2B. With reference to FIG. 3, an example scenario 300 of a warehouse 302 is shown. Warehouse 302 includes an inventory storage area 304, an order packaging area 306 and a server room 308. Warehouse 302 also includes the identification and planning system 102 associated with a warehouse 302 selection station. The exemplary warehouse 302 may also include additional operating and non-operating systems, for example, power management system, inventory management system, order processing system, inventory stores, packing stations and the like. A description of different operating and non-operating systems has been omitted from the disclosure for the sake of brevity.
[0059] In the stock storage area 304, a plurality of stock storage systems 310A is shown. . . , 310N, a robotic manipulator 312, a plurality of pallets of items of
Petition 870190138461, of 12/23/2019, p. 28/76
26/59 stock 314 and a plurality of mobile robots 316. The plurality of stock storage systems 310A ,. . . , 310N may include storage boxes that may be initially empty. A placement plan can be generated by the control server 104 to collect different stock items from one or more pallets from the plurality of stock item pallets 314 and place the selected stock items in storage compartments allocated to the selected stock items.
[0060] According to one embodiment, different stock items (for example, stored as pallets) can be allocated to different storage compartments of the plurality of 310A stock storage systems. . . , 310N, based on an intelligent allocation scheme. In the smart allocation scheme, inventory items that are in high demand (based on historical order information and / or current order information) or that may have a higher demand based on demand forecasts, forecasting methods or supply chain sources, can be filled first in the plurality of 310A stock storage systems. . . , 310N.
[0061] According to another modality, the allocation of inventory items can be decided based on a pipeline of orders that still need to be processed by the warehouse 302 service and distribution centers. More specifically, only specific inventory items for orders can initially be stored in storage compartments available from the plurality of 310A stock storage systems. . . , 310N and other storage compartments can be replenished with other inventory items, such as frequently ordered inventory items.
[0062] The robotic manipulator 312 can also be part of a delivery system that can be configured to refill different storage compartments from the plurality of 310A stock storage systems. . . , 310N with a plurality of
Petition 870190138461, of 12/23/2019, p. 29/76
27/59 stock. Robotic manipulator 312 may include a robotic arm (for example, robotic arm 118) and an end effector (for example, end effector 120, such as a gripper or a gripping tool). The functions of the robotic manipulator 312 may be similar to those of robotic manipulator 116 of the identification and planning system 102. However, the robotic manipulator 312 can perform placement operations, in which an inventory item can be selected from a storage unit, like a pallet, and positioned in a storage compartment allocated to the stock item. Thus, instead of selecting stock items in the storage compartments and taking them out to shelves or bags, the robotic manipulator 312 can perform put-back operations that are described in more detail, for example, in FIGs. 6A and 6B.
[0063] Inventory storage area 304 may further include the plurality of mobile robots 316, which can be controlled based on instructions received from one or more control servers in server room 308. Based on requirements, the plurality of mobile robots can be configured to lift and transfer one or more stock storage systems (for example, two different stock storage systems 318) from the plurality of stock storage systems 310A ... 310N, from the stock storage area 304 to an area associated with the identification and planning system 102.
[0064] In the identification and planning system 102, a first mobile robot 320A and a second mobile robot 320B can be configured to place a first stock storage system 322A and a second stock storage system 322B in specified positions in a row defined (represented by dotted lines). A scanner assembly 324 and a robotic manipulator 326 can be located in distinct positions specified in the defined queue. A 328 box may be present in the vicinity and within reach of the robotic arm of the
Petition 870190138461, of 12/23/2019, p. 30/76
28/59 robotic manipulator 326 to store automatically selectable items and manually selectable items (for example, through order consolidation) from the first 322A stock storage system and the second 322B stock storage system. The one or more control servers (for example, a control server 330) in server room 308 can be configured to perform a scan operation followed by a check operation on the inventory items of the first 322A storage system and the second 322B stock storage system, respectively. The scan operation is performed before the selection operation to avoid delays in the selection of inventory items, while in conventional solutions, the execution of the selection operation can be kept on hold while the scan operation is performed. This can lead to an increase in the response time and in the movement of a robotic manipulator in a conventional system. From the selection station, a third mobile robot 320C can be configured to transport storage stock items removed (or sorted) in box 328 to the order 306 packaging area. Alternatively, a conveyor or other transfer mechanisms can be implemented to transfer the stock items in box 328 to the packaging area of order 306. Several mobile robots loaded in box 332 can arrive in the packaging area of order 306 to transfer boxes (for example, box 328) of items from inventory for a 334 packaging station. At the 334 packaging station, inventory items can be packed and processed to fulfill multiple pipeline orders throughout the supply chain system.
[0065] Server room 308 can be a centralized (or decentralized / distributed) server room, which can include control server 330 deployed to partially or fully control the operations of the identification and planning system 102. The server room server 308 can also include a support server 336 to control
Petition 870190138461, of 12/23/2019, p. 31/76
29/59 remaining operations associated with guided movement of mobile robots, creation of stock profile, placement operations in the area of stock storage or operations related to packaging. Warehouse 302 is an exemplary warehouse that represents a specific warehouse setup, in which the proposed identification and planning system 102 can be deployed to achieve the desired throughput and cycle time for selecting and removing inventory items. However, the scope of the disclosure may not be as limited and the identification and planning system 102 can be integrated and / or implemented with different types of warehouses, regardless of size, weight, order traffic or a type of merchandise handled by them. deposits.
[0066] FIG. 4A illustrates an exemplary scanner set system and an exemplary stock storage system placed in front of the scanner set for detecting stock items from the stock storage system, according to a disclosure modality. FIG. 4A is explained in conjunction with the elements of FIGs. 1, 2A, 2B and 3. Referring to FIG. 4A, a view 400A of a scanner assembly 402 and a stock storage system 404 mounted as a payload on a mobile robot 406 is shown.
[0067] The scanner set 402 can be a 3D camera system for detecting and identifying different stock items in their respective positions on the stock storage system 404. The set of scanner 402 includes a first image sensor unit 408A and a second image sensor unit 408B that can be attached (or removably attached) to a orientable portion, i.e., a first horizontal support plate 410A and a second horizontal support plate 410B, respectively. The first image sensor unit 408A and the second image sensor unit 408B can be driven along different directions, such as vertical and horizontal directions, using an orientation mechanism (not shown) and orientable portions. The mechanism
Petition 870190138461, of 12/23/2019, p. 32/76
30/59 orientation can be a programmable or computer controlled actuator. Likewise, the orientable portion may include an actuator belt, the first horizontal support plate 410A and the second horizontal support plate 410B. The first horizontal support plate 410A and the second horizontal support plate 410B can be mounted on a vertical support structure 412.
[0068] For example, the first image sensor unit 408A and the second image sensor unit 408B can slide in the horizontal direction (left and right) on the first horizontal support plate 410A and the second horizontal support plate 410B, respectively , based on a belt that is driven by an actuator (for example, a linear actuator or a rotary actuator). Alternatively, the first image sensor unit 408A and the second image sensor unit 408B can slide in the vertical (top and bottom) direction along the vertical support structure 412.
[0069] The first image sensor unit 408A and the second image sensor unit 408B may include a first 3D camera and a second 3D camera enclosed in a first portion of the protection bracket and a second portion of the protection, respectively. A first calibration camera 414A and a second calibration camera 414B can also be attached to the first protection portion and the second protection portion, respectively. The first calibration camera 414A and the second calibration camera 414B can be attached at an angle with the first protective support portion and the second protective portion, so that the robotic manipulator (such as the robotic manipulator 116) can be in position. a field of view (FOV) of the first 414A calibration camera and the second 414B calibration camera.
[0070] As shown as an example, the stock storage system 404 may include a plurality of storage compartments 416. At any given time in the
Petition 870190138461, of 12/23/2019, p. 33/76
31/59 scanning operation, the first 408A image sensor unit may be present in front of a first 416A storage compartment and the second 408B image sensor unit may be present in front of a second 416B storage compartment. The first storage compartment 416A and the second storage compartment 416B can accommodate a first set of stock items 418 and a second set of stock items 420, respectively. Each inventory item in the first set of inventory items 418 and the second set of inventory items 420 can be organized in different ways, such as stacking side-by-side stock items or vertical stacking stock items. Stock items in the first set of stock items 418 or in the second set of stock items 420 can be homogeneous (or the same type of stock items, for example, brand A milk packs) or non-homogeneous stock items that can belong to a common product category or multiple product categories. For example, the first set of stock items 418 or the second set of stock items 420 can be dairy products of the same brand (that is, visually indistinguishable as a different product) or a mixture of packages of biscuits and bottles of vegetable oil (ie visually distinguishable as a different product).
[0071] At a specific time period, the first 3D camera and the second 3D camera can be configured to capture 3D images of a first stock item 418A and a second stock item 420A in the first storage compartment 416A and the second compartment storage system 416B, respectively, of the stock storage system 404. The position of the first 3D camera and the second 3D camera in the scanner assembly 402 at the time of scanning the first stock item 418A and the second stock item 418B may correspond to a position of the first stock item 418A and the second stock item 418B, respectively, in the
Petition 870190138461, of 12/23/2019, p. 34/76
32/59 404 stock storage.
[0072] Control server 104 can be configured to detect the first stock item 418A in the first storage compartment 416A, based on an image processing technique. Examples of the image processing technique may include, but are not limited to, such as model matching, object detection based on Invariant Resource Invariant Scale (SIFT), Accelerated Robust Resources (SURF), Independent Binary Robust Elementary Resources (BRIEF) , Quick Oriented and Rapid Rotation (ORB), Binary Robust Invariable Scalable Key Points (BRISK) or SKU identification. The control server 104 can be further configured to determine a first set of position parameters (x, y, z values) for the first stock item 418A in relation to a position of the first 3D camera. The first set of position parameters (x, y, z values) can be determined based on the registration of a position (values (x, y)) of the first 3D camera and the distance (z) by which the focal plane of a first 3D camera is separated from a point on the first stock item 418A.
[0073] The first calibration camera 414A and the second calibration camera 414B can be configured to capture images from robotic manipulator 116 at known position values (ie x '', y '', z '' and θ) from first 414A calibration camera and / or the second 414B calibration camera. The control server 104 can be configured to estimate and record a relative difference between the position values of the robotic manipulator 116 and the known position values (i.e., x '', y '', z '' and θ) of the first calibration camera 414A or the second 414B calibration camera. The recorded relative difference can be used to transform the first set of position parameters (x, y, z) into a second set of position parameters (ie, x ', y', z ', spin (r'), yaw (y '') and tilt (p ')), where rotation, yaw and tilt correspond to the axes of rotation of the robotic arm (for example, the robotic arm 118) in 3D space. The first set
Petition 870190138461, of 12/23/2019, p. 35/76
33/59 of position parameters (values of x, y, z) can be spatially different from the second set of position parameters (ie, x ', y', z ', r', y '' and p ') . This spatial differentiation can be caused by the difference in the position of the scanner assembly 402 and the robotic manipulator 116. The operation of the control server 104 for trajectory selection planning, verification and execution of the selection operation is further described in detail, for example , in FIGS. 5A and 5B.
[0074] FIG. 4B illustrates an exemplary robotic manipulator that performs a selection operation on different inventory items in an inventory storage system based on an ideal trajectory selection plan, according to a disclosure modality. FIG. 4B is explained together with the elements of FIGs. 1, 2A, 2B, 3 and 4A. With reference to FIG. 4B, a view 400B of a robotic manipulator 422 is shown. Robotic manipulator 422 may include a base support portion 424, a robotic arm 426 and an end effector 428.
[0075] The base support portion 424 includes a housing 430 that accommodates a guide rail 432 and a carriage 434 mounted on the guide rail 432. A linear actuator mechanism (not shown) can be installed in the housing 430 to move linearly the carriage 434 back and forth (for example, in a straight line) along the guide rail 432. The control server 104 can be configured to transmit instructions related to the movement of the carriage 434 along the guide rail 432. The carriage 434 also supports a column 436. The carriage 434 is attached to a first distal end 438A of the column 436 and the robotic arm 426 is mounted to a second distal end 438B of the column 436, as shown in an example.
[0076] The robotic manipulator 422 further includes a plurality of actuators 440A, 440B, 440C, 440D, 440E and 440F in the robotic arm 426. The plurality of actuators 440A, 440B, 440C, 440D, 440E and 440F can be present between two portions arm placed
Petition 870190138461, of 12/23/2019, p. 36/76
34/59 consecutively of a plurality of arm portions 442 on the robotic arm 426. The plurality of actuators 440A, 440B, 440C, 440D, 440E and 440F may allow movement of different parts of the arm of the plurality of arm portion 442 along of a defined number of degrees of freedom, such as six degrees of freedom (represented by arrows and a dotted axis line). Alternatively, each actuator of the plurality of actuators 440A, 440B, 440C, 440D, 440E and 440F can be a rotary actuator that can be activated separately to rotate a coupled arm portion along an axis of rotation (ie one of a swing, yaw, or tilt) while keeping other arm parts static.
[0077] At a free end 444 of robotic arm 426 (that is, an arm portion coupled to a 440F actuator), an automatic tool changer (not shown) can be coupled to the arm portion at the free end 444. The server Control 104 can be configured to control the automatic tool changer to reduce cycle times, automatically switching between different types of end effectors adaptively based on the detection of a type of stock item or a stock profile of an item inventory in the sweep operation.
[0078] As shown, end effector 428 can be a pneumatic clamp or holding tool with a suction cup 446 and a clamp arm 448 attached to the free end 444 of the arm portion. The end effector 428 can be a specific type of clamp suitable for a type of stock item accommodated in the stock storage system 404 (shown in FIG. 4A). The operation of robotic manipulator 422 is further described in detail, for example, in FIGs. 5A, 5B, 6A and 6B.
[0079] FIG. 5A illustrates a first scenario for sequentially executing a scan operation, a selection operation and an order consolidation for inventory items accommodated in an inventory storage system, according to a disclosure modality. FIG. 5A is explained in conjunction with the elements of FIGs. 1, 2A, 2B, 3, 4A
Petition 870190138461, of 12/23/2019, p. 37/76
35/59 and 4B. With reference to FIG. 5A, a first 500A scenario is shown.
[0080] In the first scenario 500A, a queuing station 502 is shown that includes a defined queue 504. The defined queue includes grid regions 506A, 506B, 506C, ..., 506N. Each grid region of grid regions 506A, 506B, 506C, ..., 506N can be associated with a physical space allocated in the warehouse (such as warehouse 302). The physical space can be marked by a 508 fiducial marker. A first stock storage system 510A can be moved from a stock storage area (such as the stock storage area 304) to a selection area. In the selection area, a set of scanner 512 and a robotic manipulator 514 are installed to perform a scan operation and a selection operation on different stock items in the first stock storage system 510A. The scanner assembly 512 and the robotic manipulator 514 can be functionally similar to the scanner assembly 402 and the robotic manipulator 422 of FIGs. 4A and 4B, respectively.
[0081] As shown, in an exemplary case, the scanner assembly 512 can be positioned in front of a first grid region 506A of the grid regions 506A, 506B, 506C. . . and 506N and the robotic manipulator 514 can be positioned in front of a second grid region 506B of the grid regions 506A, 506B, 506C, ..., 506N.
[0082] Control server 104 (which includes control circuit 208) can be configured to perform a queuing operation on the first stock storage system 510A before executing the selection operation. In queuing operation, control server 104 can be configured to control a first mobile robot 516A to position the first stock storage system 510A in defined queue 504 of queuing station 502. Alternatively, the first mobile robot 516A can be configured to position the first 510A stock storage system in different regions of the grid
Petition 870190138461, of 12/23/2019, p. 38/76
36/59 based on detection of fiducial marker 508. Fiducial marker 508 can act as a reference marker for the first mobile robot 516A to correct the alignment and positioning of the first stock storage system 510A in front of the scanner assembly 512 or robotic manipulator 514.
[0083] In queuing operation, control server 104 can be further configured to control the first mobile robot 516A to move the first stock storage system 510A across grid regions 506A, 506B, 506C ,. . . , 506N of defined queue 504. Alternatively, the first stock storage system 510A can be moved from one grid region to another in a sequence that depends on a position of scanner set 512 and robotic manipulator 514. The first system stock storage system 510A can be moved to perform the collection operation on a first stock item stored within the first stock storage system 510A in the defined queue 504 after the scan operation is completed and a first selection plan is selected for the robotic manipulator 514.
[0084] The first stock storage system 510A may include a first plurality of stock items 518 accommodated in storage compartments of the first stock storage system 510A. Each of the first plurality of inventory items 518 can be associated with a unique identifier that corresponds to an inventory maintenance system (SKU). The SKU for each inventory item can be a unique, physically readable, electronically readable code, or both by a plurality of image sensors 520 from the 512 scanner assembly. For example, the SKU can be a barcode, a radio frequency tag ( RF) printable or attachable, a QR code and the like. Prior to the scan operation, control server 104 can be configured to segregate the first plurality of inventory items 518 into a first set of inventory items 522 and a second
Petition 870190138461, of 12/23/2019, p. 39/76
37/59 set of stock items 522. The segregation of the first plurality of stock items can be done based on the unique tag (or SKU) for each of the first plurality of stock items 516. As shown, the first set inventory item 522 includes a first inventory item 522A (that is, an automatically selectable inventory item) and the second set of inventory items 524 includes a second inventory item 524A manually selectable by a warehouse operator 526.
[0085] Initially, the first stock storage system 510 may be present in the first region of grid 504A (for example, above the fiducial marker 506). The control server 104 can be configured to perform, using the plurality of image sensors 520 of the scanner assembly 512, the scanning operation on the first plurality of stock items 518 on the first stock storage system 510A. In the scanning operation, the plurality of image sensors 520 can be configured to capture a plurality of 3D images of the first plurality of stock items 516. Each 3D image can include RGB-D information about one or more stock items in the field. viewing an image sensor from the plurality of image sensors 520. The RGB-D information can be used by the control server 104 to detect the first plurality of stock items 518 in the first stock storage system 510A.
[0086] In order to select the first stock item 522A, the control server 104 can be configured to extract color information and depth information from a 3D image that includes the first stock item 522A. The 3D information can be estimated by the control server 104 for the first stock item 522A. The 3D information can correspond to different surface attributes of the first stock item 522A in the first stock storage system 510A, which can be obtained from a scan of the first plurality of stock items 516. Different surface attributes can
Petition 870190138461, of 12/23/2019, p. 40/76
38/59 can be estimated to identify optimal position values and different parameters (for example, grip values, pressure and a type of clamp). Optimal position values and different parameters may be required by the robotic manipulator 514 to select the first 522A stock item with less chance of falling or damaging the first 522A stock item during selection and removal of the first 522A stock item in bags or shelves, like shelf 528.
[0087] Control server 104 can be further configured to determine a first set of position parameters for a first stock item 522A from the first detected plurality of stock items 516, with respect to a position of an image sensor the plurality of image sensors 520 in the scanner set 512. The first set of position parameters can correspond to the position values of the first stock item 522A in terms of 3D Cartesian coordinates. The first two coordinates can be represented by the xy position values of the image sensor in front of the first stock item 522A and a third coordinate can be represented by a value of distance z between the focal plane of the image sensor and a point on the first stock item 522A.
[0088] During the movement of the first stock storage system 510A, different stock items may be affected by vibrations, traction forces or pushing forces due to sudden turns, changes in elevation, (abrupt) changes in the speed of movement or other factors. Such factors can cause non-uniform orientation or misalignment of stock items between them in different storage compartments of the first stock storage system 510A. This can also cause an error in determining appropriate capture parameters (for example, gripping parameters and position parameters) for robotic manipulator 514, due to the uncertainty (as shown in FIG. 4A) in the orientation of different inventory items. Like this,
Petition 870190138461, of 12/23/2019, p. 41/76
39/59 the control server 104 can be configured to estimate an orientation of the first stock item 522A (and / or other self-selectable stock items) in the first stock storage system 510A. The estimated orientation (for example, 30 degrees tilted to the right from a vertical) can be used to adjust the appropriate capture parameters for a robotic arm (for example, the robotic arm 426) and an end effector (for example , the end effector 428) of the robotic manipulator 514. This can further assist in the precise maneuvering of the end effector to reach, hold and remove the first stock item 522A in a bag or on a shelf, such as shelf 528.
[0089] The control server 104A can be configured to collect and store a set of attributes of the first stock item 522A before the scan operation is performed. For example, the set of attributes of the first stock item 522A can be collected and stored at the time of performing a placement operation to replenish empty storage compartments of the first stock storage system 510A. In the scan operation, the control server 104 can be further configured to retrieve the set of attributes associated with the first stock item 522A. The set of attributes can include, without limitation, size, a Stock Keeping Unit (SKU) identifier, a shape, a price, a degree of fragility, a type of packaging, a type of product material, a related item order, constraint specified by the customer or supply chain for the first 522A inventory item. The control server 104 can be further configured to estimate a set of gripping parameters associated with the end effector of the robotic manipulator 514, based on the retrieved set of attributes from the first stock item 522A. Examples of the set of gripping parameters may include, but are not limited to, suction cup grip strength / pressure, number of suction cups, type of gripper (suction cup, magnetic gripper or pneumatic gripper or actuator hands) and a
Petition 870190138461, of 12/23/2019, p. 42/76
40/59 clamp size.
[0090] To select the first 522A stock item, an ideal trajectory selection plan may be required to specify different parameters related to the movement of the robotic arm and end effector for the first 522A stock item. The control server 104 can be configured to generate a plurality of path selection plans for the first stock item 522A. Each path selection plane of the plurality of path selection plans can correspond to a transformation of the first set of position parameters (x, y, z) into a second set of position parameters (x1, y1, z ₁ , rotation, yaw, tilt) in relation to the robotic manipulator 514 (ie, an origin of the robotic manipulator 514). The plurality of trajectory selection planes can be generated by applying a plurality of transformation operations on the first set of position parameters (x, y, z) to obtain the second set of position parameters (x ₁ , y ₁ , z ₁ , turn, yaw, tilt). According to one embodiment, the plurality of transformation operations can correspond to iterative changes in the first set of position parameters (that is, position values measured in relation to an image sensor) based on the estimated 3D surface information and the estimated orientation of the first inventory item. The control server 104 can use the retrieved set of attributes as supplementary information to the 3D information obtained from the 3D scans of the different stock items to generate the plurality of trajectory selection plans.
[0091] In order to select an ideal path selection plan (ie, a first path selection plan), from the plurality of path selection plans generated, a test computer simulation for each path selection plan the plurality of trajectory selection plans can be performed by the control server 104. The test computer simulation can be viewed even more in the
Petition 870190138461, of 12/23/2019, p. 43/76
41/59
HMI 122 that can be connected communicatively to the control server 104, through communication network 124. In the test computer simulation, the control server 104 can be configured to verify a successful selection operation corresponding to the application of each control plan. selection of the generated plurality of trajectory selection plans. In an implementation, the success of a selection operation can be verified by simulating the execution of the selection operation corresponding to the application of each transformation operation from the plurality of transformation operations in the first determined set of position parameters. Alternatively, in each transformation operation, the position parameters for the robotic manipulator 514 can be adjusted and rendered in a simulation to verify that the robotic arm and end effector are able to reach and hold the first stock item 522A of the first stock storage system 510A. In addition, in the test computer simulation, the success of the selection operation of the first stock item 522A can also be verified by executing the selection operation with a plurality of modifications in the estimated set of gripping parameters.
[0092] As an example, in the simulation, the gripping parameters can be pressure values that can be modified in a recurring way. It can also be checked whether the end effector (for example, a pneumatic clamp) is able to successfully grab and hold the first item in stock for a necessary period, without falling or damaging the packaging or product, referred to as the first stock item 522A. In the test simulation, the success of the selection operation can correspond to an event in the test computer simulation, in which a simulation of the robotic manipulator 514 chooses the first stock item 522A using a simulated movement of the plurality of actuators (for example, the plurality of actuators 440A, 440B, 440C, 440D, 440E and 440F). The simulated movement can be based on a defined number of degrees of freedom of the arm
Petition 870190138461, of 12/23/2019, p. 44/76
42/59 robotic and in the second plurality of position parameters of a path selection plane (for example, an ideal path selection plane).
[0093] The control server 104 can be further configured to select a first path selection plan from the plurality of path selection plans generated as an ideal path selection plan. The first path selection plane can be selected before a selection operation is performed on the first 522A stock item. The selection of the first path selection plane can be based on the success of the selection operation in the test computer simulation. Success can be verified based on the second set of position parameters corresponding to the first path selection plane.
[0094] In the selection operation, the control server 104 can be further configured to control the robotic arm and end effector of the robotic manipulator 514, to select the first stock item 522A of the first stock storage system 510A, with based on the first trajectory selection plane. More specifically, in the selection operation, the robotic manipulator 514 can further control a plurality of actuators on the robotic arm to allow movement of the robotic arm and end effector within the defined number of degrees of freedom in 3D space. The defined number of degrees of freedom can be six degrees of freedom together with the horizontal displacement of the robotic manipulator 514 along a guide rail (for example, the guide rail 432).
[0095] According to an embodiment, the control server 104 can be further configured to check whether order consolidation is necessary in the cycle (that is, a cycle for selecting inventory items for an order or a batch of orders). The check for order consolidation can be performed before the selection operation (for example, at the time of the scan operation on different stock items in the first
Petition 870190138461, of 12/23/2019, p. 45/76
43/59 stock storage system 510A). In order consolidation, inventory items, segregated in the inventory storage area based on different attributes, such as fragility, price or shape, into manually selectable inventory items, can be removed by warehouse workers (such as warehouse operator 526 ) and placed on bags or shelves, such as shelf 528. In certain embodiments, if order consolidation is required, control server 104 can be configured to display information on various pick-put-to-light (PPTL) devices (not shown) placed in the storage compartments of the first 510A stock storage system. Information (for example, SKU identifier) can be displayed to facilitate the selection, removal and targeted sorting of manually selectable inventory items from a third 506C grid region of defined queue 504. This can help the operator to minimize errors, speed up cycle and improve order processing accuracy, as well as improving overall warehouse worker productivity.
[0096] FIG. 5B illustrates a second scenario for the scheduled execution of the scanning operation and the selection operation for a row of different stock storage systems, according to a disclosure modality. FIG. 5B is explained in conjunction with the elements of FIGs. 1, 2A, 2B, 3, 4A, 4B and 5A. With reference to FIG. 5B, a second 500B scenario is shown.
[0097] In the second scenario 500B, queuing station 502 is shown which includes the defined queue 504. The defined queue includes grid regions 506A, 506B ..... 506N. The first stock storage system 510A and a second stock storage system 510B can be moved by the first mobile robot 516A and a second mobile robot 516B, respectively, from a stock storage area (such as the stock storage area 304 ) for a selection area. In the collection area, the scanner set 512 and robotic manipulator 514 can be
Petition 870190138461, of 12/23/2019, p. 46/76
44/59 installed to perform the scan operation and selection operation. As an example case, scanner set 512 can be positioned in front of a first grid region 506A of grid regions 506A, 506B ,. . . , 506N and robotic manipulator 514 can be positioned in front of a second grid region 506B from grid regions 506A, 506B, ..., 506N.
[0098] Before executing the scan operation, control server 104 (which includes control circuit 208) can be configured to perform a queuing operation on the first stock storage system 510A and the second stock storage system 510B. In queuing operation, control server 104 can be configured to control the first mobile robot 516A and the second mobile robot 516B to position the first stock storage system 510A and the second stock storage system 510B, respectively, in the queue set 504 of queuing station 502. Alternatively, the first mobile robot 516A can be configured to position the first stock storage system 510A and the second stock storage system 510B in the first regions of grid 506A and the second region of grid 506B , respectively, based on detection of fiducial marker 508. Fiducial marker 508 can act as a reference marker for the first mobile robot 516A and the second mobile robot 516B to correct the alignment and positioning of the first stock storage system 510A and the second stock storage system 510B, respectively, in front of the 512 scanner assembly or the robotic manipulator 514.
[0099] According to one embodiment, the control server 104 can be configured to perform the selection operation on the first stock item 522A of the first stock storage system 510A in parallel with the scan operation on a second plurality of items stock 530 in one or more storage compartments
Petition 870190138461, of 12/23/2019, p. 47/76
45/59 storage of the second 510B stock storage system. The first stock storage system 510A and the second stock storage system 510B can be moved to perform the selection operation on the first stock item 522A in parallel with the scan operation on the second plurality of stock items 530 stored in the second 510B stock storage system.
[0100] The selection operation on the first stock item 522A of the first stock storage system 508A and the scan operation on the second plurality of stock items 530 of the second stock storage system 510B can be performed the first time (T1 ) and a second time (T ₂ ), respectively. The first time (T1) and the second time (T2) can correspond to a parallelism (T1 equal to T2) in the execution of the scan operation and the selection operation for the first stock storage system 510A and the second storage system of stock 510B. Alternatively, the first time (T1) and the second time (T2) may correspond to an asymmetry (partial overlap or not) in the initialization time (T1 is not equal to T2) of the scan operation and the selection operation for the first 510A stock storage system and the second 510B stock storage system.
[0101] FIGS. 6A and 6B, collectively, illustrate a scenario for the execution of a placement operation by an exemplary robotic manipulator in two different stages, according to a disclosure modality. FIGs. 6A and 6B are explained together with the elements of FIGs. 1, 2A, 2B, 3, 4A, 4B, 5A and 5B. With reference to FIGs. 6A and 6B, a scenario 600 is shown. In scenario 600, a robotic manipulator 602 (which may correspond to robotic manipulator 422 of FIG. 4B), a stock storage system 604 and a box 606 in a storage area are shown. inventory (such as stock storage area 304) of a warehouse (such as warehouse 302).
[0102] Scenario 600 represents an operation of
Petition 870190138461, of 12/23/2019, p. 48/76
46/59 placement, in which robotic manipulator 602 can be configured to select inventory items from a plurality of inventory items 608 arranged in box 606 (or, in some cases, a pallet) and place selected inventory items in compartments (from stock storage system 604) allocated to each stock item of the plurality of stock items 608. The placement operation can be part of a replenishment process to fill empty storage compartments with stock items that you may need be withdrawn in subsequent selection cycles. In some embodiments, the control server 104 can be configured to generate a placement plan that can describe a position of individual SKUs (i.e., stock items) in storage positions of the stock storage system 604.
[0103] In FIGs. 6A and 6B, robotic manipulator 602 can be configured to select a first stock item 610 from the plurality of stock items 608 in box 606 and place the first stock item 610 in a storage compartment 612 of the stock storage system 604 The movement of the robotic arm and the automatic selection and alteration of the final effectors can be done based on instructions from the control server 104.
[0104] FIGS. 7A, 7B and 7C collectively illustrate a flow chart that describes a method for fulfilling orders in a warehouse, according to a disclosure modality. FIGs. 7A, 7B and 7C are explained together with the elements of FIGs. 1, 2A, 2B, 3, 4A, 4B, 5A, 5B, 6A and 6B. With reference to FIGs. 7A, 7B and 7C, a flow chart 700 is shown. The operations described in flow chart 700 can start at 702 and proceed to 704.
[0105] In 704, order information associated with one or more orders can be received from an order management system. Control server 104 can be configured to receive order information associated with one or more orders from the
Petition 870190138461, of 12/23/2019, p. 49/76
47/59 order management.
[0106] In 706, a list of inventory items to be selected to fulfill one or more orders received can be generated. The control server 104 can be configured to generate the list of inventory items to be selected to fulfill the one or more orders received.
[0107] In 708, different inventory items in the generated list can be segregated into automatically selectable inventory items and manually selectable inventory items. Control server 104 can be configured to segregate the different inventory items in the list generated into automatically selectable inventory items and manually selectable inventory items.
[0108] In 710, a queuing operation can be performed on a first stock storage system (for example, the first stock storage system 106A) and a second stock storage system. The control server 104 can be configured to perform the queuing operation on the first inventory storage system and the second inventory storage system. An example of the queuing operation is shown and described in FIG. 5A.
[0109] In 712, in the queuing operation, one or more mobile robots can be controlled to position the first inventory storage system and the second inventory storage system in a defined queue of a queuing station. In queuing operation, control server 104 can be configured to control one or more mobile robots to position the first inventory storage system and the second inventory storage system in a defined queue of a queuing station (for example , queuing station 502).
[0110] In 714, a scan operation can be
Petition 870190138461, of 12/23/2019, p. 50/76
48/59 performed, using a plurality of image sensors 114, on a first plurality of inventory items in the first inventory storage system (e.g., the first inventory storage system 106A). The control server 104 can be configured to perform the scanning operation, using the plurality of image sensors 114, on the first plurality of stock items in the first stock storage system 106A. An example of the scan operation is shown and described in FIGs. 4A and 5A.
[0111] In 716, the first plurality of stock items in the first stock storage system 106A can be detected using the plurality of image sensors 114. The control server 104 can be configured to detect the first plurality of stock items in the first stock storage system 106A using the plurality of image sensors 114.
[0112] In 718, color information and depth information can be extracted from the plurality of 3D images of the plurality of stock items. The control server 104 can be configured to extract color information and depth information from the plurality of 3D images from the plurality of stock items.
[0113] In 720, 3D information corresponding to different surface attributes of a first stock item can be estimated based on scanning the first plurality of stock items by the plurality of image sensors 114. Control server 104 can be configured to estimate 3D information that corresponds to different surface attributes of a first inventory item, based on scanning the first plurality of inventory items by the plurality of image sensors 114.
[0114] At 722, an orientation of the first stock item in the first stock storage system 106A can be estimated based on the scan of the first plurality of stock items
Petition 870190138461, of 12/23/2019, p. 51/76
49/59 by the plurality of image sensors 114. The control server 104 can be configured to estimate the orientation of the first stock item in the first stock storage system 106A based on scanning the first plurality of stock items for the plurality of 114 image sensors.
[0115] In 724, a first set of position parameters for the first stock item can be determined from the first plurality of stock items detected, with respect to the position of an image sensor of the plurality of image sensors 114 in the scanner set 112. Control server 104 can be configured to determine the first set of position parameters for the first stock item from the first plurality of stock items detected, with respect to the position of a plurality image sensor image sensors 114 in scanner assembly 112.
[0116] In 726, a plurality of transformation operations can be applied to the first set of position parameters determined to obtain a second set of position parameters, based on the estimated 3D surface information and the estimated orientation of the first stock item . The control server 104 can be configured to apply the plurality of transformation operations on the first set of position parameters determined to obtain a second set of position parameters, based on the estimated 3D surface information and the estimated orientation of the first item of the stock.
[0117] In 728, for the first inventory item, a plurality of trajectory selection plans can be generated, in which each trajectory selection plan of the plurality of trajectory plans corresponds to the transformation of the first set of determined position parameters for the second set of position parameters in relation to the robotic manipulator 116. The control server 104 can be configured to generate, for the first stock item, the plurality of selection plans
Petition 870190138461, of 12/23/2019, p. 52/76
50/59 trajectory, in which each trajectory selection plane of the plurality of trajectory selection planes corresponds to the transformation of the first set of position parameters determined to the second set of position parameters in relation to the robotic manipulator 116.
[0118] In 730, in a test computer simulation, a success of the selection operation corresponding to the plurality of trajectory selection plans generated can be verified for the first stock item. The control server 104 can be configured to verify, in the test computer simulation, a success of the selection operation corresponding to the plurality of trajectory selection plans generated can be verified for the first stock item.
[0119] At 732, a set of attributes of the first inventory item can be stored before the scan operation. Control server 104 can be configured to store the attribute set of the first inventory item prior to the scan operation.
[0120] In 734, a set of adhesion parameters associated with end effector 120 of robotic manipulator 116 can be estimated based on the set of attributes stored in the first stock item. Control server 104 can be configured to estimate the set of adhesion parameters associated with end effector 120 of robotic manipulator 116 based on the stored set of attributes of the first stock item.
[0121] In 736, in the test computer simulation, a success of the selection operation corresponding to the plurality of trajectory selection plans generated for the first stock item can be verified based on the execution of the selection operation with a plurality of changes in the set of estimated adherence parameters. The control server 104 can be configured to verify, in the test computer simulation, a success of the selection operation corresponding to the plurality of trajectory selection plans generated for the first item of
Petition 870190138461, of 12/23/2019, p. 53/76
51/59 inventory, based on the execution of the selection operation with a plurality of modifications in the set of estimated adhesion parameters.
[0122] In 738, a first trajectory selection plan can be selected from the plurality of trajectory selection plans before the selection operation is performed on the first stock item, based on the success of the selection operation in the simulation of test computer. Control server 104 can be configured to select the first path selection plan from the plurality of path selection plans before the selection operation is performed on the first stock item, based on the success of the selection operation in the simulation test computer.
[0123] At 740, end effector 120 can be selected from a set of final effectors, based on the set of attributes stored in the first inventory item. Control server 104 can be configured to control robotic manipulator 116 to select end effector 120 from the set of end effectors, based on the stored set of attributes of the first stock item.
[0124] In 742, in the selection operation, the robotic arm 118 and end effector 120 can be controlled to select the first stock item of the first stock storage system 106A, based on the first trajectory selection plane. The control server 104 is configured, in the selection operation, to control the robotic arm 118 and the end effector 120 to select the first stock item from the first stock storage system 106A, based on the first selection plane trajectory. Control goes to the end. An example of the selection operation is shown and described in FIGs. 4B and 5A.
[0125] Certain disclosure modalities can be found in an identification and planning system (for example, the identification and planning system 102) for order fulfillment. Various disclosure modalities can provide an identification system
Petition 870190138461, of 12/23/2019, p. 54/76
52/59 and planning that includes a scanner set (for example, scanner set 112), a robotic manipulator (for example, robotic manipulator 116) and a control server (for example, control server 104) that includes control circuits (for example, control circuit 208). The scanner assembly may include a plurality of image sensors (e.g., the plurality of image sensors 114) and the robotic manipulator may include a robotic arm (e.g., robotic arm 118) and an end effector (e.g., the final effector 120) detachable to the robotic arm. The control circuits can be configured to detect, by the plurality of image sensors, a first plurality of stock items in a first stock storage system (for example, the first stock storage system 106A) and determine a first set of position parameters for a first stock item of the first plurality of stock items detected. The first set of position parameters can be determined in relation to a position of the plurality of image sensors in the scanner set. The control circuits can be further configured to generate a plurality of trajectory selection plans for the first stock item. Each path selection plane of the plurality of path selection planes can correspond to a transformation from the first set of position parameters determined to a second set of position parameters in relation to the robotic manipulator. According to one embodiment, the first set of position parameters can be spatially different from the second set of position parameters. Such spatial differentiation can be caused by a difference in the position of the scanner assembly and the robotic manipulator. A first trajectory selection plane can be selected, through the control circuits, from the plurality of trajectory selection plans. The first trajectory selection plane is selected before a selection operation is performed on the first stock item. According to one modality, the control circuits can be
Petition 870190138461, of 12/23/2019, p. 55/76
53/59 configured to select the first trajectory selection plane for the first inventory item during an idle state of the robotic manipulator and the scanner assembly. The selection of the first path selection plane can be based on the success of the selection operation in a test computer simulation, based on the second set of position parameters corresponding to the first path selection plane. In the selection operation, the control circuits can be additionally configured to control the robotic arm and the final effector to select the first stock item of the first stock storage system, based on the first trajectory selection plan.
[0126] According to one embodiment, the identification and planning system may additionally include a first mobile robot (for example, the first mobile robot 110A). The first inventory storage system can be mounted on the first mobile robot that can be configured to move the first inventory storage system from a first location to a second location in a warehouse based on instructions received from the control circuits. The first stock storage system may include a first plurality of storage compartments. Each storage compartment of the first plurality of storage compartments present in different spatial positions in the first inventory storage system can accommodate the first plurality of inventory items.
[0127] According to one embodiment, the control circuits can be additionally configured to perform, using the plurality of image sensors, a scanning operation on the first plurality of stock items in the first stock storage system. In the scanning operation, the control circuits can be further configured to extract color information and depth information from a plurality of three-dimensional (3D) images from the first plurality of stock items. The scan operation can include
Petition 870190138461, of 12/23/2019, p. 56/76
54/59 in addition to detecting the first plurality of stock items and determining the first set of position parameters.
[0128] According to one embodiment, the identification and planning system may additionally include a second stock storage system which may include a second plurality of storage compartments. Each storage compartment of the second plurality of storage compartments present in different spatial positions in the second stock storage system can accommodate a second plurality of stock items. The control circuits can be configured to perform the selection operation on the first stock item of the first stock storage system in parallel with the scan operation on the second plurality of stock items in one or more storage compartments of the second stock storage system. stock storage.
[0129] According to one modality, the control circuits can be additionally configured to perform a queuing operation on the first stock storage system and a second stock storage system before executing the selection operation. The identification and planning system can additionally include at least one mobile robot. In queuing operation, the control circuits can be configured to control at least one mobile robot to position the first inventory storage system and the second inventory storage system in a defined queue of a queuing station. The defined queue can include grid regions. The scanner assembly can be positioned in front of a first grid region of the grid regions and the robotic manipulator is positioned in front of a second grid region of the grid regions. In queuing operation, the control circuits can be additionally configured to control at least one mobile robot to move the first stock storage system and the second stock storage system.
Petition 870190138461, of 12/23/2019, p. 57/76
55/59 inventory through the grid regions of the defined queue. The first stock storage system and the second stock storage system can be moved to perform the selection operation on the first stock item in parallel with the scan operation on the second plurality of stock items stored on the second stock storage system. stock.
[0130] According to one modality, the control circuits can be additionally configured to estimate 3D three-dimensional information that corresponds to different surface attributes of the first stock item in the first stock storage system, based on a scan, for the plurality image sensors, the first plurality of stock items. In addition, an orientation of the first stock item in the first stock storage system can be estimated by the control circuits based on scanning the first plurality of stock items. The control circuits can be further configured to apply a plurality of transformation operations to the first set of position parameters determined to obtain the second set of position parameters. The plurality of transformation operations can be applied based on the estimated 3D surface information and the estimated orientation of the first inventory item. In certain modalities, the control circuits can be additionally configured to verify, in the test computer simulation, the success of the selection operation of the first stock item. The success of the operation can be verified based on the execution of the selection operation corresponding to the application of each transformation operation of the plurality of transformation operations in the first set of determined position parameters.
[0131] According to one embodiment, the robotic manipulator may additionally include a plurality of actuators in the robotic arm. The plurality of actuators can allow movement of the arm
Petition 870190138461, of 12/23/2019, p. 58/76
56/59 robot and the final effector within a defined number of degrees of freedom in a 3D space. The success of the selection operation may correspond to an event in the test computer simulation, in which a simulation of the robotic manipulator selects the first stock item using a simulated movement of the plurality of actuators. The simulated movement can be based on a defined number of degrees of freedom of the robotic arm and the second plurality of position parameters in the first trajectory selection plane.
[0132] According to a modality, the identification and planning system can additionally include a memory that can be configured to store a set of attributes of the first stock item before the scanning operation and the selection operation. The control circuits can be additionally configured to estimate a set of adhesion parameters associated with the final effector of the robotic manipulator, based on the stored set of attributes of the first stock item. The control circuits can be additionally configured to verify, in the test computer simulation, the success of the selection operation of the first stock item by performing the selection operation with a plurality of modifications in the set of estimated adhesion parameters. The control circuits can be further configured to select the final effector from a set of final effectors, based on the set of stored attributes of the first stock item.
[0133] According to one modality, each of the first plurality of inventory items detected in the first inventory storage system can be associated with a unique tag that corresponds to an inventory maintenance system (SKU). The control circuits can be further configured to segregate, before the scanning operation and the selection operation, the first plurality of stock items in a first set of stock items and a second set of stock items. The segregation of the first plurality of items of
Petition 870190138461, of 12/23/2019, p. 59/76
57/59 inventory can be made based on the unique tag for each of the first plurality of inventory items. The first set of inventory items can correspond to inventory items that are automatically selectable by the robotic handler and the second set of inventory items can correspond to inventory items manually selectable by a warehouse operator.
[0134] Certain disclosure modalities can be found in an identification and planning system for order fulfillment. Various types of disclosure can provide an identification and planning system that includes a control server. The control server can include control circuits, which can be configured to detect a first plurality of inventory items in a first inventory storage system and determine a first set of position parameters for a first inventory item of the first plurality of detected inventory items. The first set of position parameters can be determined in relation to a position of a plurality of image sensors. The control circuits can be further configured to generate a plurality of trajectory selection plans for the first stock item. Each path selection plane of the plurality of path selection plans can correspond to a transformation from the first set of position parameters determined to a second set of position parameters in relation to a robotic manipulator. The control circuits can be further configured to select a first path selection plane from the plurality of path selection planes. The first trajectory selection plane is selected before a selection operation is performed on the first stock item. The selection of the first path selection plane can be made based on the success of the selection operation in a test computer simulation. The success of the selection operation can be determined based on the second set of position parameters
Petition 870190138461, of 12/23/2019, p. 60/76
58/59 corresponding to the first trajectory selection plane. In the selection operation, the control circuits can be additionally configured to control a robotic arm and a final effector detachable to the robotic arm to select the first stock item of the first stock storage system, based on the foreground plan. trajectory selection.
[0135] Although the present disclosure is described with reference to certain modalities, it should be understood by those skilled in the art that various changes can be made and equivalents can be replaced without departing from the scope of the present disclosure. In addition, many modifications can be made to adapt a situation or material to the teachings of the present disclosure without departing from its scope. Therefore, it is intended that the present disclosure is not limited to the particular modality disclosed, but that the present disclosure will include all modalities covered by the scope of the attached claims. Equivalent elements, materials, processes or steps can be replaced by those represented and described in this document. In addition, certain characteristics of the disclosure can be used regardless of the use of other characteristics, all as would be apparent to a person skilled in the art after having the benefit of this description of the disclosure.
[0136] As used in this document, the terms comprise, comprising, include, including, have, have or any contextual variants thereof, are intended to cover a non-exclusive inclusion. For example, a process, product, article or device that comprises a list of elements is not necessarily limited to just those elements, but may include other elements not expressly listed or inherent in that process, product, article or device. In addition, unless it is expressly stated otherwise, it either refers to one or inclusive and not one or exclusive. For example, a condition of A or B is
Petition 870190138461, of 12/23/2019, p. 61/76
59/59 satisfied by any of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present) and both A and B are true (or gifts).
[0137] Although the steps, operations or calculations can be presented in a specific order, that order can be changed in different ways. In some modalities, to the extent that several steps are shown as sequential in this specification, some combination of such steps in alternative modalities can be performed at the same time. The sequence of operations described in this document can be interrupted, suspended, reversed or controlled by another process. It will also be noted that one or more of the elements represented in the drawings / figures can also be implemented in a more separate or integrated manner, or even removed or rendered inoperative in certain cases, as useful according to a specific application.

权利要求:
Claims (25)
[1]
1. Identification and planning system, characterized by the fact that it comprises:
a scanner assembly comprising one or more image sensors;
a robotic manipulator comprising a robotic arm and an end effector detachably attached to the robotic arm; and a control server comprising control circuits, where the control circuits are configured to:
detecting, by one or more image sensors, a first plurality of stock items in a first stock storage system;
determining a first set of position parameters for a first stock item from the first detected plurality of stock items, with respect to the position of an image sensor of the one or more image sensors in the scanner set;
generate a plurality of trajectory selection plans for the first inventory item, where each trajectory selection plan of the plurality of trajectory plans corresponds to a transformation of the first determined set of position parameters into a second set of position parameters in relation to the robotic manipulator;
select a first path selection plan in the plurality of path selection plans, where the first path selection plan is selected before a selection operation is performed on the first inventory item, and where the first plan selection trajectory selection is based on the success of the selection operation in a test computer simulation, based on the second set of position parameters corresponding to the first selected trajectory selection plane; and
Petition 870190108721, of 10/25/2019, p. 8/17
[2]
2/9 in the selection operation, control the robotic arm and end effector to select the first stock item from the first stock storage system, based on the first trajectory selection plane.
2. Identification and planning system, according to claim 1, characterized by the fact that it additionally comprises a first mobile robot, in which the first stock storage system is mounted on the first mobile robot which is configured to move the first system storage of stock from a first location to a second location in a warehouse based on instructions received from the control circuit.
[3]
3. The identification and planning system according to claim 1, characterized by the fact that the first stock storage system comprises a first plurality of storage compartments, wherein each storage compartment of the first plurality of storage compartments, present in different spatial positions in the first stock storage system, accommodates the first plurality of stock items.
[4]
4. Identification and planning system, according to claim 1, characterized by the fact that the control circuits are additionally configured to perform, using one or more image sensors, a scanning operation on the first plurality of stock items in the first stock storage system.
[5]
5. Identification and planning system, according to claim 4, characterized by the fact that, in the scanning operation, the control circuits are additionally configured to extract color information and depth information from a plurality of three-dimensional (3D) images ) of the first plurality of inventory items, and
Petition 870190108721, of 10/25/2019, p. 9/17
3/9 in which the scanning operation further comprises the detection of the first plurality of stock items and the determination of the first set of position parameters.
[6]
6. Identification and planning system according to claim 1, characterized by the fact that it additionally comprises a second stock storage system comprising a second plurality of storage compartments, wherein each storage compartment of the second plurality of compartments of storage, present in different spatial positions in the second stock storage system, accommodates a second plurality of stock items.
[7]
7. Identification and planning system, according to claim 6, characterized by the fact that the control circuits are additionally configured to perform the selection operation on the first stock item of the first stock storage system in parallel with an operation scanning the second plurality of stock items in one or more storage compartments of the second stock storage system.
[8]
8. Identification and planning system, according to claim 1, characterized by the fact that the control circuits are additionally configured to perform a queuing operation on the first stock storage system and a second stock storage system before execution of the selection operation.
[9]
9. Identification and planning system, according to claim 8, characterized by the fact that it additionally comprises at least one mobile robot, in which, in the queuing operation, the control circuits are configured to control at least one mobile robot to position the first inventory storage system and the second
Petition 870190108721, of 10/25/2019, p. 10/17
4/9 stock storage system in a defined queue of a queuing station.
[10]
10. Identification and planning system, according to claim 9, characterized by the fact that the defined queue comprises grid regions, and in which the scanner set is positioned in front of a first grid region of the grid regions and the robotic manipulator is positioned in front of a second grid region of the grid regions.
[11]
11. Identification and planning system, according to claim 10, characterized by the fact that, in the queuing operation, the control circuits are additionally configured to control at least one mobile robot to move the first stock storage system and the second stock storage system across the grid regions of the defined queue, and where the first stock storage system and the second stock storage system are moved to perform the selection operation on the first stock item in parallel with the scan operation on the second plurality of stock items stored in the second stock storage system.
[12]
12. Identification and planning system, according to claim 1, characterized by the fact that the control circuits are additionally configured for:
estimate three-dimensional (3D) information that corresponds to different surface attributes of the first stock item in the first stock storage system, based on a scan, by one or more image sensors, of the first plurality of stock items; and estimate an orientation of the first inventory item in the first inventory storage system, based on the scan.
Petition 870190108721, of 10/25/2019, p. 11/17
5/9
[13]
13. Identification and planning system, according to claim 12, characterized by the fact that the control circuits are additionally configured to apply a plurality of transformation operations in the first set of position parameters determined to obtain the second set of parameters position, based on the estimated 3D surface information and the estimated orientation of the first inventory item.
[14]
14. Identification and planning system, according to claim 13, characterized by the fact that the control circuits are additionally configured to verify, in the test computer simulation, the success of the selection operation of the first stock item, with based on the execution of the selection operation corresponding to the application of each transformation operation of the plurality of transformation operations in the first set of determined position parameters.
[15]
15. Identification and planning system, according to claim 1, characterized by the fact that the robotic manipulator additionally comprises a plurality of actuators in the robotic arm, in which the plurality of actuators allows movement of the robotic arm and the end effector within a defined number of degrees of freedom in a three-dimensional (3D) space.
[16]
16. Identification and planning system, according to claim 15, characterized by the fact that the success of the selection operation corresponds to an event in the test computer simulation, in which a simulation of the robotic manipulator selects the first stock item using a simulated movement of the plurality of actuators, and in which the simulated movement is based on a defined number of degrees of freedom of the robotic arm and on the second plurality of position parameters in the first trajectory selection plane.
Petition 870190108721, of 10/25/2019, p. 12/17
6/9
[17]
17. Identification and planning system, according to claim 1, characterized by the fact that it additionally comprises a memory configured to store a set of attributes of the first stock item before a scanning operation and the selection operation, in which the control circuits are additionally configured to estimate a set of adhesion parameters associated with the end effector of the robotic manipulator, based on the stored set of attributes of the first stock item.
[18]
18. Identification and planning system, according to claim 17, characterized by the fact that the control circuits are additionally configured to verify, in the test computer simulation, the success of the selection operation of the first stock item by the execution selection operation with a plurality of modifications in the set of estimated adherence parameters.
[19]
19. Identification and planning system, according to claim 18, characterized by the fact that the control circuits are additionally configured to select the end effector from a set of final effectors, based on the set of stored attributes of the first item of stock.
[20]
20. Identification and planning system, according to claim 1, characterized by the fact that each of the first plurality of inventory items detected in the first inventory storage system is associated with a unique tag that corresponds to a unit of stock maintenance (SKU).
[21]
21. Identification and planning system, according to claim 20, characterized by the fact that the control circuits are additionally configured to segregate, before a scanning operation and selection operation, the first plurality of stock items
Petition 870190108721, of 10/25/2019, p. 13/17
7/9 on a first set of inventory items and a second set of inventory items.
[22]
22. Identification and planning system, according to claim 21, characterized by the fact that the segregation of the first plurality of stock items is based on the unique tag for each of the first plurality of stock items, and in which the first set of inventory items corresponds to inventory items that are automatically selectable by the robotic handler and the second set of inventory items corresponds to inventory items manually selectable by a warehouse operator.
[23]
23. Identification and planning system, according to claim 1, characterized by the fact that the first set of position parameters is spatially different from the second set of position parameters and in which the spatial differentiation is caused by a difference in one spatial location of the scanner set and robotic manipulator.
[24]
24. Identification and planning system characterized by the fact that it comprises:
a control server comprising control circuits, where the control circuits are configured to:
detecting a first plurality of stock items in a first stock storage system;
determining a first set of position parameters for a first stock item from the first detected plurality of stock items, with respect to the position of an image sensor of one or more image sensors;
generate a plurality of trajectory selection plans for the first inventory item, in which each trajectory
Petition 870190108721, of 10/25/2019, p. 14/17
8/9 plurality of trajectory planes corresponds to a transformation of the first set of position parameters into a second set of position parameters in relation to a robotic manipulator;
select a first path selection plan in the plurality of path selection plans, where the first path selection plan is selected before a selection operation is performed on the first inventory item and where the selection of the first path plan trajectory selection is based on the success of the selection operation in a test computer simulation, based on the second set of position parameters corresponding to the first trajectory selection plane; and in the selection operation, control the robotic arm and the end effector detachably connected to the robotic arm to select the first stock item of the first stock storage system, based on the first trajectory selection plane.
[25]
25. Method of an identification and planning system that includes control circuits and one or more image sensors, characterized by the fact that the method comprises:
detecting, by one or more image sensors, a first plurality of stock items in a first stock storage system;
determine, by the control circuits, a first set of position parameters for a first stock item from the first detected plurality of stock items, with respect to the position of an image sensor of one or more image sensors;
generate, through the control circuits, a plurality of trajectory selection plans for the first stock item, in which each trajectory selection plan of the plurality of trajectory plans corresponds to a transformation of the first set of determined
Petition 870190108721, of 10/25/2019, p. 15/17
9/9 position parameters in a second set of position parameters in relation to a robotic manipulator;
select, through the control circuits, a first trajectory selection plane in the plurality of trajectory selection plans, in which the first trajectory selection plane is selected before a selection operation is performed on the first stock item and in which the selection of the first path selection plane is based on the success of the selection operation in a test computer simulation, based on the second set of position parameters corresponding to the first path selection plane; and in the selection operation, control, through the control circuits, the robotic arm and the end effector detachably connected to the robotic arm to select the first stock item of the first stock storage system, based on the first selection plane trajectory.

类似技术:

---

公开号 | 公开日 | 专利标题

---

BR102019022468A2|2020-05-26|IDENTIFICATION AND PLANNING SYSTEM AND METHOD FOR FILLING ORDERS

---

US10272566B2|2019-04-30|Robotic grasping of items in inventory system

---

AU2016328637B2|2019-07-25|Networked robotic manipulators

---

US9926138B1|2018-03-27|Determination of removal strategies

---

CN109279252A|2019-01-29|Cargo movement system and method

---

US10766149B2|2020-09-08|Optimization-based spring lattice deformation model for soft materials

---

JP2020196624A|2020-12-10|Robotic system for palletizing packages by using real time placement simulation

---

US11077554B2|2021-08-03|Controller and control method for robotic system

---

JP2020196625A|2020-12-10|Robotic system for processing packages arriving out of sequence

---

CN111498212B|2021-05-07|Robotic system for handling out-of-order arriving packages

---

US10958895B1|2021-03-23|High speed automated capture of 3D models of packaged items

---

US11065761B2|2021-07-20|Robotic picking training technique

---

EP3728084B1|2021-07-07|Method for palletising heterogeneous objects

---

US10919151B1|2021-02-16|Robotic device control optimization using spring lattice deformation model

---

CN209550002U|2019-10-29|Logistics object handles system and sorting device

---

US20220081216A1|2022-03-17|Presort system for executing robot-assisted putaway tasks

---

US20210395014A1|2021-12-23|Handling robot, material fetching method, material replenishing or returning method, intelligent warehousing system

---

US20220081215A1|2022-03-17|Dynamic item putaway management using mobile robots

---

WO2020021307A1|2020-01-30|Robotic picking training technique

同族专利:

---

公开号 | 公开日

---

JP2020066534A|2020-04-30|

---

CN111105181A|2020-05-05|

---

CA3059313A1|2020-04-25|

---

EP3643456A2|2020-04-29|

---

KR20200047433A|2020-05-07|

---

SG10201910062PA|2020-05-28|

---

US10752442B2|2020-08-25|

---

US20200130936A1|2020-04-30|

---

EP3643456A3|2020-07-29|

---

AU2019250278A1|2020-05-14|

---

US10399778B1|2019-09-03|

---

CL2019003046A1|2020-03-27|

引用文献:

---

公开号 | 申请日 | 公开日 | 申请人 | 专利标题

---

---

JPH02256430A|1989-03-29|1990-10-17|Mitsubishi Electric Corp|Automatic assembly device and method|

---

WO1991011885A1|1990-01-29|1991-08-08|Technistar Corporation|Automated assembly and packaging system|

---

SE1050763A1|2010-07-08|2010-07-12|Abb Research Ltd|A method for calibration of a mobile robot|

---

JP5911299B2|2011-12-27|2016-04-27|キヤノン株式会社|Information processing apparatus, information processing apparatus control method, and program|

---

EP3135445B1|2014-03-28|2021-04-28|Sony Corporation|Robot arm device|

---

US9733646B1|2014-11-10|2017-08-15|X Development Llc|Heterogeneous fleet of robots for collaborative object processing|

---

US10076840B2|2015-04-03|2018-09-18|Canon Kabushiki Kaisha|Information processing apparatus, information processing method, and program|

---

JP6458713B2|2015-11-18|2019-01-30|オムロン株式会社|Simulation device, simulation method, and simulation program|

---

US9694494B1|2015-12-11|2017-07-04|Amazon Technologies, Inc.|Feature identification and extrapolation for robotic item grasping|

---

KR101868374B1|2016-10-20|2018-06-18|엘지전자 주식회사|Control method of a moving-robotf|US10148918B1|2015-04-06|2018-12-04|Position Imaging, Inc.|Modular shelving systems for package tracking|

---

US11120392B2|2017-01-06|2021-09-14|Position Imaging, Inc.|System and method of calibrating a directional light source relative to a camera's field of view|

---

JP2020078853A|2018-11-14|2020-05-28|東芝テック株式会社|Commodity ejection device|

---

WO2020146861A1|2019-01-11|2020-07-16|Position Imaging, Inc.|Computer-vision-based object tracking and guidance module|

---

US11004033B1|2019-02-04|2021-05-11|Vecna Robotics, Inc.|System and method of asynchronous and automated order fulfillment|

---

EP3953282A1|2019-04-08|2022-02-16|Lineage Logistics, LLC|Automatic transportation of pallets of goods|

---

US10846812B1|2019-09-25|2020-11-24|Coupang Corp.|Computer-implemented systems and methods for centralized logistics monitoring|

---

GB2588227A|2019-10-18|2021-04-21|Grey Orange Pte Ltd|Method and system for handling object pallets in storage facilities|

---

US11164226B2|2019-11-01|2021-11-02|AiFi Inc.|Method and system for managing product items in a store|

---

KR20210129433A|2020-04-20|2021-10-28|에스케이하이닉스 주식회사|Storage saving method for big data processing|

---

WO2021245991A1|2020-06-01|2021-12-09|パナソニックＩｐマネジメント株式会社|Sorting work assistance method, program, transport control system, and transport system|

---

EP3950232A1|2020-08-04|2022-02-09|Bayerische Motoren Werke Aktiengesellschaft|Method for testing a container being capable to store at least one vehicle component|

---

CN112046993B|2020-10-15|2022-02-01|北京京东振世信息技术有限公司|Goods warehousing processing method, device and equipment and goods warehousing system|

---

CN112340347A|2020-10-26|2021-02-09|北京极智嘉科技有限公司|Goods shelving system and method|

法律状态:
2020-05-26| B03A| Publication of a patent application or of a certificate of addition of invention [chapter 3.1 patent gazette]|

优先权:

---

申请号 | 申请日 | 专利标题

---

US16/170,958|2018-10-25|

---

US16/170,958|US10399778B1|2018-10-25|2018-10-25|Identification and planning system and method for fulfillment of orders|

---